The present invention relates to single-plate color solid-state image pickup apparatus in which the resolution characteristics of hue and luminance are not debased when all the pixels are read out and yet color generation can be performed by a single line where lines are skipped (thinned out) when read out.
Color filter array constructed as shown in FIGS. 1A, 1B, 1C are known as using complementary color filters of the generally used type to produce color signals when all pixels of the image-pickup device are sequentially read out in a single-plate color solid-state image pickup apparatus. In this color filter array, four color filters Ye, Mg, Cy and G are arranged in the manner of a mosaic. FIG. 1A shows the signal processing mode for producing CR signal; FIG. 1B shows the signal processing mode for producing CB signal; and FIG. 1C shows the signal processing mode for producing YL signal. It should be noted that CR signal corresponds to (Rxe2x88x92Y) color-difference signal, CB to (Bxe2x88x92Y) color-difference signal, and YL to Y signal. Color signals (RGB) can be produced by combining these three signals. The numerals 1, 2, 3 and 4 in FIG. 1B correspond to Ye, Mg, Cy and G, respectively, representing their order of arrangement which may be considered as the like of the array shown in FIGS. 1A and 1C. The complementary color filters Ye, Mg and Cy are the filters for indicating the components of Ye=R+G, Mg=R+B and Cy=B+G, respectively. Further, the CR signal, CB signal and YL signal can be expressed as in the following equations (1), (2) and (3):
CR=(Ye+Mg)xe2x88x92(Cy+G)=2Rxe2x88x92Gxe2x80x83xe2x80x83(1)
CB=(Cy+Mg)xe2x88x92(Ye+G)=2Bxe2x88x92Gxe2x80x83xe2x80x83(2)
YL=Ye+Mg+Cy+G=2R+3G+2Bxe2x80x83xe2x80x83(3)
A description will now be given with respect to the mode of producing CR signal as shown in FIG. 1A. For the first pixel of the first line of FIG. 1A, CR signal of the first pixel can be produced by the signal processing of (Ye11+Mg21)xe2x88x92(Cy12+G22). CR signal of the second pixel is produced by the signal processing of (Ye13+Mg23)xe2x88x92(Cy12+G22). By sequentially forming CR signals of the pixels of third and after in a similar manner, all the CR signals for the first line can be provided onto the line. At the first pixel of the third line, CR signal of the first pixel can be produced in a similar manner as in the first line by the signal processing of (Ye32+Mg42)xe2x88x92(Cy31+G41). A similar signal processing can be performed for the second and third pixels of the third line and also for other odd-number lines to produce CR signals onto all the odd-number lines. CR signal of the even-number lines, for example of the first pixel of the second line, can be produced by interpolating between the CR signals of above and below, i.e., the first pixel of the first line and the first pixel of the third line. It should be noted that, in the figures, an interpolation for producing a signal is represented by its abbreviation xe2x80x9cINTxe2x80x9d and the direction of such interpolation is indicated by arrow. CR signals corresponding to other even-number lines can be also produced by vertically interpolating in a similar manner. Production is thus possible of the CR signals corresponding to all the pixels. Further, the CB signals and YL signals can be produced by performing signal processing as shown in FIGS. 1B and 1C.
Shown in FIG. 2 is a block diagram of a digital circuit for processing signal to produce CR, CB and YL signals, the operation of which will be described below with reference to FIGS. 1A, 1B, 1C. First, addition of lines (vertical addition) is performed by adding together and averaging input signal SigIN and a signal obtained by delaying input signal SigIN by 1H (line). In particular, the line addition is sequentially performed for example as first line+second line, second line+third line. Next, to produce YL signal (FIG. 1C), signal A derived from such line addition and signal B derived from delaying of signal A by 1D (data) are added together and averaged to obtain a horizontally added signal which becomes YL signal.
To produce CR signal (FIG. 1A) and CB signal (FIG. 1B) signal (Axe2x88x92B) derived from subtraction from the line added signal A of signal B which is obtained by delaying signal A by 1D (data), and signal (Bxe2x88x92A) derived from the inverted subtraction thereof are alternately selected for each 1D (data). Thus obtained signal is CR/CB signal. The CR/CB becomes CR signal for the odd-number lines and CB signal for the even-number lines. Accordingly, a signal obtained by adding together and averaging (vertically interpolating) CR/CB signal and one derived from 2H-delay of CR/CB signal, and a signal obtained by delaying CR/CB signal by 1H can be alternately selected for each 1H to form CR signal and CB signal.
A description will be given below by way of FIG. 3 with respect to a known system for producing color signals by using a mixed complementary color filter when all pixels of the image-pickup device are sequentially read out similarly as the above. A mixed complementary color filter array is formed by arranging four color filters Wr, Gr, Gb and Wb in the manner of a mosaic as shown in FIGS. 3A, 3B, 3C. FIG. 3A shows the signal processing mode for producing CR signal; FIG. 3B shows the signal processing mode for producing CB signal; and FIG. 3C shows the signal processing mode for producing YL signal. It should be noted that a disclosure has been made in Japanese Patent Publication No.Hei-1-42192 with respect to the mixed complementary color filter.
The numerals 1, 2, 3 and 4 in FIG. 3B correspond to Wr, Gr, Gb and Wb, respectively, representing their order of arrangement which may be considered as the like of the array shown in FIGS. 3A and 3C. The mixed complementary color filters Wr, Gr, Gb and Wb are the filters possessing the components of Wr=Ye+Mg, Gr=Ye+G, Gb=Cy+G and Wb=Cy+Mg, respectively. Further, from Ye=R+G, Mg=R+B and Cy=B+G, the signals CR, CB and YL can be expressed by the following equations (4), (5) and (6):
CR=(Ye+Mg)xe2x88x92(Cy+G)=Wrxe2x88x92Gb=2Rxe2x88x92Gxe2x80x83xe2x80x83(4)
CB=(Cy+Mg)xe2x88x92(Ye+G)=Wbxe2x88x92Gr=2Bxe2x88x92Gxe2x80x83xe2x80x83(5)
YL=Ye+Mg+Cy+G=Wr+Gb=Wb+Gr=2R+3G+2Bxe2x80x83xe2x80x83(6)
A description will now be given with respect to the mode of producing CR signal as shown in FIG. 3A. For the first pixel of the first line of FIG. 3A, CR signal of the first pixel can be produced by the signal processing of (Wr11xe2x88x92Gb12). CR signal of the second pixel is produced by the signal processing of (Wr13xe2x88x92Gb12). By sequentially forming the CR signals of the pixels of third and after in a similar manner, all the CR signals of the first line can be provided onto the line. At the first pixel of the third line, CR signal of the first pixel can be produced in a similar manner as in the first line by the signal processing of (Wr32xe2x88x92Gb31). By performing a similar signal processing for the second and third pixels of the third line and also for other odd-number lines, CR signal can be provided onto all the odd-number lines. CR signal of the even-number lines, for example of the first pixel of the second line, can be produced by interpolating between the CR signals of above and below, i.e., the first pixel of the first line and the first pixel of the third line. The CR signals corresponding to other even-number lines can be also produced by vertical interpolation in a similar manner. Production is thus possible of the CR signals corresponding to all the pixels. Further, the CB signals and YL signals can be produced by performing signal processing as shown in FIGS. 3B and 3C.
Shown in FIG. 4 is a block diagram of a digital circuit for processing signal to produce CR, CB and YL signals when using the above-described mixed complementary filter array, the operation of which will be described below with reference to FIGS. 3A, 3B and 3C. This digital signal processing circuit omits the first line addition (vertical addition) processing part in the digital signal processing circuit of the case of forming color signals using the complementary color filter array shown in FIG. 2. First, to produce YL signal, a horizontally added signal is obtained by adding together and averaging input signal SigIN (signal A) and signal B derived from delaying of signal A by 1D (data). This becomes YL signal [FIG. 3C].
To produce CR signal (FIG. 3A) and CB signal (FIG. 3B) signal (Axe2x88x92B) derived from subtraction from input signal (SigIN)A of signal B which is obtained by delaying the signal A by 1D (data), and signal (Bxe2x88x92A) derived from the inverted subtraction thereof are alternately selected for each 1D (data). Thus obtained signal is CR/CB signal. The CR/CB becomes CR signal for the odd-number lines and CB signal for the even-number lines. Accordingly, a signal obtained by adding together and averaging (vertically interpolating) CR/CB signal and one derived from 2H-delay of CR/CB signal, and a signal obtained by delaying CR/CB signal by 1H can be alternately selected for each 1H to form CR signal and CB signal.
A description will be given below by way of FIGS. 5A, 5B, 5C with respect to a conventional system for producing color signals by using RGB primary color filters when all pixels of the image-pickup device are sequentially read out similarly as-the above. An RGB primary color filter array as shown in FIGS. 5A, 5B, 5C is the color filter array of the so-called Bayer arrangement. FIG. 5A shows the processing mode for producing R signal; FIG. 5B shows the processing mode for producing B signal; and FIG. 5C shows the processing mode for producing G signal. The numerals 1, 2, 3 and 4 in FIG. 5B correspond to R, B, G and G, respectively, representing their order of arrangement which may be considered as the like of the array shown in FIGS. 5A and 5C.
A description will now be given with respect to the mode of producing R signal as shown in FIG. 5A. In the case of producing R signal, the signals corresponding to filters R11, R13, R31 and R33 are extracted in their unmodified state, and interpolations in right-and-left and in up-and-down directions are performed based on these R signals to produce all the R signals. B signals are produced also by performing a similar processing as shown-in FIG. 5B. G signals, too, are produced by performing similar interpolations as shown in FIG. 5C, though G filter outnumbers others. R, G, B signals can be produced by the above processing.
Shown in FIG. 6 is a block diagram of a digital circuit for processing signal to produce color signals using the above-described RGB primary color filter array, the operation of which will be described below with reference to FIGS. 5A, 5B, 5C. First, to produce R signal (FIG. 5A) and B signal (FIG. 5B), interpolation between lines in up and down direction is possible by selecting for each 1H from a signal obtained by adding together and averaging input signal SigIN and signal derived from 2H-delay of SigIN, and a signal obtained by delaying the input signal SigIN by 1H. Thereby, R/G signal and B/G signal are formed. A signal obtained by adding together and averaging such R/G signal and signal derived from delaying of R/G signal by 2D (data), and a signal obtained by delaying R/G signal by 1D (data) are alternately selected for each 1D to produce R signal. Similarly, a signal obtained by adding together and averaging B/G signal and signal derived from delaying of B/G signal by 2D, and a signal obtained by delaying B/G signal by 1D are alternately selected for each 1D to produce B signal. In respect of G signal, a signal obtained by delaying input signal SigIN by 1H and 1D, and a signal obtained by adding together and averaging a signal obtained by adding together and averaging input signal SigIN and signal derived from 2H-delay of the input signal SigIN and a signal obtained by delaying such signal by 2D are alternately selected for each 1D to produce G signal.
Generally, when emphasis is on the resolution in constructing the color filter array of a solid-state image pickup device in a single-plate color solid-state image pickup apparatus, one based on a color line-by-line sequential system such as the above-described color filter arrays is advantageous. However, since a greater number of pixels are used in constructing the recent solid-state image pickup apparatus such as those in the electronic cameras using CCD imaging device, the pixels thereof are subject to a skip readout and not all of them are read out except for the case of recording a still image. In particular, they are thinned out when read out to correspond for example to the performing of AE/AF operation and displaying of an image on a liquid crystal display unit which require a high-speed processing.
If, for example, every other line is simply thinned out when read out by applying a skip readout system to a single-plate color solid-state image pickup apparatus using a color filter array of the color line-by-line sequence system, it is impossible to obtain data consisting of a line-by-line sequence. For example, if a thinned-out readout is performed by omitting every other line when a complementary color filter array of the structure as shown in FIGS. 1A, 1B, 1C is employed, there is a case where only the components of CR signal can be extracted, as CR signal from the first line, CR signal from the third line and CR signal from the fifth line, too. In this case, CB signal cannot be extracted, resulting in a problem that color signals cannot be formed within one frame.
Such problem occurs also in the case of using the mixed complementary color filter array shown in FIGS. 3A, 3B, 3C, or using the primary color filter array shown in FIGS. 5A, 5B, 5C. In particular, CR signal and CB signal are alternately outputted for example in the order of CR, CB, CR, CB in the color signal forming method using the mixed complementary color filter array shown in FIGS. 3A, 3B, 3C. Similarly to the case of using the complementary color filter array, therefore, this results in only CR signal or only CB signal being outputted when a skip readout of every other line is performed, making it impossible to produce color signals within one frame. Further, in the case of using the primary color filter array shown in FIGS. 5A, 5B, 5C, a thinned-out readout of every other line may results in a state where only the components of R, G signals can be extracted as R, G from the first line, R, G from the third line and R, G from the fifth line, too. It becomes impossible to extract B signal and color signals cannot be formed within one frame.
In view of the above problems with the single-plate color solid-state image pickup apparatus using known color filter arrays, it is an object of the present invention to provide a single-plate solid-state color image pickup apparatus using a color filter array capable of deriving color signals from a single line in the case where lines are skipped (thinned out) when read out, without debasing the resolution characteristics of hue and luminance when all the pixels are read out.
In accordance with a first aspect of the present invention, there is provided a single-plate color solid-state image pickup apparatus including a solid-state image pickup-device having a plurality of pixels arrayed in horizontal and vertical directions, and a color filter array consisting of a plurality of color filters arrayed in horizontal and vertical directions correspondingly to the respective pixels of the solid-state image pickup device. The color filter array comprises complementary color filters, constructed by sequentially arranging color filter columns in horizontal direction in a repeating cycle of four pixels of first, second, third, and fourth. First color-difference signals are obtained as modulated signals periodically in a certain number of pixels from xe2x80x9cfirst+secondxe2x80x9d and xe2x80x9cthird+fourthxe2x80x9d pixel columns, and second color-difference signals are obtained as modulated signals periodically in the same pixel number as the first color-difference signals from xe2x80x9csecond+thirdxe2x80x9d and xe2x80x9cfourth+fifthxe2x80x9d pixel columns, wherein the first and second color-difference signals obtained from xe2x80x9cthird+fourthxe2x80x9d and xe2x80x9cfourth+fifthxe2x80x9d pixel columns are different respectively in phase by 180 degrees from the first and second color-difference signals obtained from xe2x80x9cfirst+secondxe2x80x9d and xe2x80x9csecond+thirdxe2x80x9d pixel columns.
Further, in accordance with a second aspect of the invention, the single-plate color solid-state image pickup apparatus of the first aspect includes control means for driving and controlling said solid-state image pickup device, the control means having a drive function of the mode for recording a still image by extracting pixel signals of all the pixels by sequentially scanning said solid-state image pickup device and a drive function of the mode for recording a still image or for performing a dynamic image processing by extracting pixel signals of every n (n being an integer of 1 or greater) lines out of every m (m being an integer of 2 or greater) lines in the vertical direction from said solid-state image pickup device.
Furthermore, in accordance with a third aspect of the invention, the single-plate color solid-state image pickup apparatus of the second aspect, when performing the operation of recording a still image or processing a dynamic image by extracting pixel signals of n lines out of every m lines in the vertical direction from said solid-state image pickup device, is constructed to generate color signals by each single line by means of a row of color filters of 4-pixel repeating cycle of first, second, third and fourth in the horizontal direction.
By using a color filter array consisting of complementary color filters arranged as the above, the resolution characteristics of hue and luminance are not debased when all the pixels are read out without increasing the rate of operating clock, and color signals can be generated by a single line when lines are skipped in readout. It is therefore possible to avoid a deviation in colors no matter how the lines are skipped when read out.
In accordance with a fourth aspect of the present invention, there is provided a single-plate color solid-state image pickup apparatus including a solid-state image pickup device having a plurality of pixels arrayed in horizontal and vertical directions, and a color filter array consisting of a plurality of color filters arrayed in horizontal and vertical directions correspondingly to the respective pixels of the solid-state image pickup device. The color filter array comprises mixed complementary color filters, constructed by sequentially arranging color filter columns in horizontal direction in a repeating cycle of four pixels of first, second, third, and fourth. First color-difference signals are obtained as modulated signals periodically in a certain number of pixels from first and third pixel columns, and second color-difference signals are obtained as modulated signals periodically in the same pixel number as the first color-difference signals from second and fourth pixel columns, wherein the first and second color-difference signals obtained from third and fourth pixel columns are different respectively in phase by 180 degrees from the first and second color-difference signals obtained from first and second pixel columns.
Further, in accordance with a fifth aspect of the invention, the single-plate color solid-state image pickup apparatus of the fourth aspect includes control means for driving and controlling said solid-state image pickup device, the control means having a drive function of the mode for recording a still image by extracting pixel signals of all the pixels by sequentially scanning said solid-state image pickup device and a drive function of the mode for recording a still image or for performing a dynamic image processing by extracting pixel signals of every n (n being an integer of 1 or greater) lines out of every m (m being an integer of 2 or greater) lines in the vertical direction from said solid-state image pickup device.
Furthermore, in accordance with a sixth aspect of the invention, the single-plate color solid-state image pickup apparatus of the fifth aspect, when performing the operation of recording a still image or processing a dynamic image by extracting pixel signals of n lines out of every m lines in the vertical direction from said solid-state image pickup device, is constructed to generate color signals by each single line by means of a row of color filters of 4-pixel repeating cycle of first, second, third and fourth in the horizontal direction.
By using a color filter array consisting of mixed complementary color filters arranged as the above, the resolution characteristics of hue and luminance are not debased when all the pixels are read out. Since, furthermore, color signals can be generated by a single line when lines are skipped in readout, a deviation in colors is not caused no matter how the lines are skipped when read out. Further, human eyes are vested with a higher resolution in horizontal direction than in the vertical direction. Accordingly, the color line-by-line sequential system where luminance signal is generated by adding together pixels adjacent to each other in horizontal direction is disadvantageous in that it causes the resolution in horizontal direction to be degraded. In the above described fourth, fifth and sixth aspects of the present invention, however, the horizontal resolution can be improved comparing to that of using the mixed complementary color filters of the known color line-by-line sequential system, since pixels adjacent to each other in the vertical direction are added together to generate a luminance signal.
In accordance with a seventh aspect of the present invention, there is provided a single-plate color solid-state image pickup apparatus including a solid-state image pickup device having a plurality of pixels arrayed in horizontal and vertical directions, a color filter array consisting of color filters of three kinds different one another in spectral sensitivity, arrayed in horizontal and vertical directions correspondingly to the respective pixels of the solid-state image pickup device, and control means for driving and controlling the solid-state image pickup device, the control means having a drive function of the mode for recording a still image by extracting pixel signal of all the pixels through a sequential scan of the solid-state image pickup device and a drive function of the mode for recording a still image or processing a dynamic image by extracting pixel signals of every n (n being an integer of 1 or greater) lines out of every m (m being an integer of 2 or greater) lines in the vertical direction from the solid-state image pickup device. The color filter array is consisting of primary color filters, each row of the color filters being arranged in the order of the color filters of first, second, third and third in the horizontal direction, wherein the color filter row of n+1""th line is shifted by 2-pixel pitch in the horizontal direction in relation to the color filter row of n""th line.
Furthermore, in accordance with a eighth aspect of the invention, the single-plate color solid-state image pickup apparatus according to the seventh aspect of the invention, when performing the operation of recording a still image or processing a dynamic image by extracting pixel signals of n lines out of every m lines in the vertical direction from the solid-state image pickup device, is constructed to generate color singles by each single line by means of a row of color filters of 4-pixel repeating cycle of first, second, third and fourth in the horizontal direction.
By using a color filter array consisting of primary color filters arranged as the above, similarly to the cases of the above described first to sixth aspects, the resolution characteristics of hue and luminance are not debased when all the pixels are read out. Since, furthermore, color signals can be generated by a single line when lines are skipped in readout, deviation in colors is not caused no matter how the lines are skipped when read out.